-
Notifications
You must be signed in to change notification settings - Fork 13.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide #10999
Conversation
I will duplicate files for the later Chinese translation before merging and reference old model (#11004). |
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 3ed9750 (Mon Feb 03 13:34:16 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
CI report:
Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @azagrebin. I finished the first part of my review (general documentation) and had some comments. I will continue with the remaining parts.
docs/ops/memory/mem_setup.md
Outdated
The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink | ||
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. | ||
|
||
<strong>Note: This memory setup guide is relevant only for Task Executors!</strong> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Inconsistent casing. Sometimes Task Executor is written with capital letters and sometimes with lower case letters. I'd suggest to stick to one way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here I wanted to emphasise that it is for TM not JM. I can try to use closer to TM.
docs/ops/memory/mem_setup.md
Outdated
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. | ||
|
||
<strong>Note: This memory setup guide is relevant only for Task Executors!</strong> | ||
Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same for "Job Manager"
docs/ops/memory/mem_setup.md
Outdated
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. | ||
|
||
<strong>Note: This memory setup guide is relevant only for Task Executors!</strong> | ||
Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager. | |
Check [Job Manager related configuration options](../config.html#jobmanager) for the memory setup of Job Manager. |
docs/ops/memory/mem_setup.md
Outdated
a bigger JVM limit in this case. | ||
|
||
<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed | ||
that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation. | |
to never exceed its configured size. Therefore, resizing network memory will not necessarily help in this situation. |
docs/ops/memory/mem_setup.md
Outdated
|
||
See also [detailed Memory Model](#detailed-memory-model). | ||
|
||
## Detailed Memory Model |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest to make this a separate page. This will shorten the page a bit and make it easier to digest for the reader.
f2ec29a
to
8cb09c8
Compare
under the License. | ||
--> | ||
|
||
* toc |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Can you move the TOC below the first paragraph, I think it looks nicer
docs/ops/memory/mem_tuning.md
Outdated
--> | ||
|
||
* toc | ||
{:toc} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, please move to line 30 below the first paragraph
docs/ops/memory/mem_tuning.md
Outdated
[Managed memory](../memory/mem_setup.html#managed-memory) helps Flink to run the batch operators efficiently. | ||
Therefore, the size of the [managed memory](mem_setup.html#managed-memory) can affect the performance of [batch jobs](../../dev/batch). | ||
If it makes sense, Flink will try to allocate and use as much *managed memory* as configured for batch jobs but not beyond its limit. | ||
It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations. | ||
If Flink runs out of [managed memory](../memory/mem_setup.html#managed-memory), it utilizes disk space. | ||
Using [managed memory](../memory/mem_setup.html#managed-memory), some operations can be performed directly | ||
on the raw data without having to deserialize the data to convert it into Java objects. All in all, | ||
[managed memory](../memory/mem_setup.html#managed-memory) improves the robustness and speed of the system. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Flink's batch operators leverage managed memory to run more efficiently. In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. This means that managed memory configurations have practical effects on the performance of your applications. Flink will attempt to allocate and use as much managed memory as configured for batch jobs but not go beyond its limits. This prevents
OutOfMemoryException
's because Flink knows precisely how much memory it has to leverage. If the managed memory is not sufficient, Flink will gracefully spill to disk.
@azagrebin I have a few small comments on the placement of the TOC and other minutiae. I'm relying on Till and others to validate this for correctness but overall I think this looks good. |
09865b1
to
4ff8243
Compare
Thanks for the review @sjwiesman ! I have addressed the comments. |
ok, please tag me if anything is dramatically changed and you’d like me to check the language otherwise tentative +1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for opening the PR, @azagrebin.
I've also left some comments, most of them related to the links.
In addition, since state_backends.zh.md
is already translated into Chinese, I translated the new added contents in this file in my comments. Maybe @carp84 or @Myasuka can help verify my translation.
docs/ops/memory/mem_setup.md
Outdated
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. | ||
|
||
<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>! | ||
Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems the linked page does not have the anchor #jobmanager
. This link jumps to the beginning of the configuration page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, the docs have been recently restructured. I will find a new link
docs/ops/memory/mem_setup.md
Outdated
* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) | ||
* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These two links do not work for me. Why do we have -1
in the urls?
Is it a mistake or something wrong with my building?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The options, which have -1
at the end of their links, were duplicated in the main configuration page. They were listed as common options and then as TM or TM mem specific. After docs restructure, they got deduplicated. I will remove -1
for them.
docs/ops/memory/mem_setup.md
Outdated
[Here](#detailed-memory-model) are more details about the other memory components. | ||
|
||
Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory | ||
is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap. | |
is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory](#managed-memory), and direct memory. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to align the description with the figure above, to avoid potential confusing.
docs/ops/memory/mem_setup.md
Outdated
* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) | ||
* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same problem of '-1' in urls for these two lines.
docs/ops/memory/mem_setup.md
Outdated
This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory). | ||
|
||
<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail. | ||
This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not necessarily in flink-conf.yaml. It can also be '-D' parameters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, I will just remove in *flink-conf.yaml*
docs/ops/state/state_backends.md
Outdated
@@ -74,6 +74,8 @@ The MemoryStateBackend is encouraged for: | |||
- Local development and debugging | |||
- Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer requires very little state. | |||
|
|||
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. | |
It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero. |
docs/ops/state/state_backends.zh.md
Outdated
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. | ||
This will ensure that the maximum amount of memory is allocated for user code on the JVM. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Translation:
建议同时将 [managed memory](../memory/mem_setup.html#managed-memory) 设为0,以保证将最大限度的内存分配给 JVM 上的用户代码。
docs/ops/state/state_backends.md
Outdated
@@ -92,6 +94,9 @@ The FsStateBackend is encouraged for: | |||
- Jobs with large state, long windows, large key/value states. | |||
- All high-availability setups. | |||
|
|||
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. | |
It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero. |
docs/ops/state/state_backends.zh.md
Outdated
It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero. | ||
This will ensure that the maximum amount of memory is allocated for user code on the JVM. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Translation:
建议同时将 [managed memory](../memory/mem_setup.html#managed-memory) 设为0,以保证将最大限度的内存分配给 JVM 上的用户代码。
docs/ops/state/state_backends.zh.md
Outdated
@@ -115,6 +120,8 @@ RocksDBStateBackend 的适用场景: | |||
然而,这也意味着使用 RocksDBStateBackend 将会使应用程序的最大吞吐量降低。 | |||
所有的读写都必须序列化、反序列化操作,这个比基于堆内存的 state backend 的效率要低很多。 | |||
|
|||
Check also recommendations about the [task executor memory configuration](../memory/mem_tuning.html#rocksdb-state-backend) for the RocksDBStateBackend. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Translation:
请同时参考 [Task Executor 内存配置](../memory/mem_tuning.html#rocksdb-state-backend) 中关于 RocksDBStateBackend 的建议。
docs/ops/memory/mem_tuning.md
Outdated
In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. | ||
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | ||
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | ||
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryException`'s because Flink knows precisely |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The class name is OutOfMemoryError
Thanks for the review @tillrohrmann |
bac3201
to
16eeb75
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing my comments, @azagrebin. LGTM.
In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. | ||
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | ||
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | ||
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Plural is without '
just s
Maybe it's easier to just write OutOfMemoryErrors
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sjwiesman what's your opinion? is it typo?
docs/ops/memory/mem_migration.md
Outdated
|
||
This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink. | ||
|
||
The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why no backticks here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, I overlooked old options w/o links
docs/ops/memory/mem_migration.md
Outdated
This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink. | ||
|
||
The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size) | ||
in the default ‘flink-conf.yaml’. The value is also increased from 1024Mb to 1568Mb. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here as well, why no backticks? There are a couple of these.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for updating this PR @azagrebin. I had some additional comments.
docs/ops/memory/mem_setup.md
Outdated
The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink | ||
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: In the first sentence we use italics for the release version and in the next sentence we don't use italics.
from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release. | ||
|
||
<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>! | ||
Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager. | |
Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of a job manager. |
|
||
The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*) | ||
and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap, | ||
*managed memory* (managed by Flink) and other direct (or native) memory. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should direct memory be italic similar to the other components?
<br /> | ||
|
||
If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration | ||
options are relevant, see also [local execution](mem_detail.html#local-execution) for more details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
options are relevant, see also [local execution](mem_detail.html#local-execution) for more details. | |
options are relevant. See [local execution](mem_detail.html#local-execution) for more details. |
If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration | ||
options are relevant, see also [local execution](mem_detail.html#local-execution) for more details. | ||
|
||
Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options: | |
Otherwise, the simplest way to setup memory in Flink is to configure either of the following two options: |
<br/> | ||
|
||
All of the components listed above can be but do not have to be explicitly configured for the local execution. | ||
If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and | |
If they are not configured they are set to their default values. [*Task heap memory*](mem_setup.html#task-operator-heap-memory) and |
|
||
All of the components listed above can be but do not have to be explicitly configured for the local execution. | ||
If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and | ||
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory) | |
*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [*managed memory*](mem_setup.html#managed-memory) |
has a default value of 128Mb only for the local execution mode. | ||
|
||
<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case. | ||
It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started | |
The actual JVM heap size of the started |
<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case. | ||
It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started | ||
local process is not controlled by Flink and depends on how you start the process. | ||
If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. *-Xmx*, *-Xms*. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. *-Xmx*, *-Xms*. | |
If you want to control the JVM heap size, then you have to explicitly pass the corresponding arguments *-Xmx* and *-Xms* to the JVM. |
docs/ops/memory/mem_migration.md
Outdated
@@ -0,0 +1,234 @@ | |||
--- | |||
title: "Migration from old configuration (before 1.10 release)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not consistent capitalization wrt the other titles.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
title: "Migration from old configuration (before 1.10 release)" | |
title: "Migration Guide" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some more comments.
|
||
## IllegalConfigurationException | ||
|
||
If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates | |
If you see an `IllegalConfigurationException` thrown from `TaskExecutorProcessUtils`, it usually indicates |
@@ -0,0 +1,86 @@ | |||
--- | |||
title: "Memory tuning guide" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
title: "Memory tuning guide" | |
title: "Memory Tuning Guide" |
under the License. | ||
--> | ||
|
||
In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors | |
In addition to the [main memory setup guide](mem_setup.html), this section explains how to configure the task executor's memory for different workloads and deployments. |
--> | ||
|
||
In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors | ||
depending on the use case and which options are important in which case. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
depending on the use case and which options are important in which case. |
* toc | ||
{:toc} | ||
|
||
## Configure memory for standalone deployment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Capitalization of heading is not consistent with the headings of the other pages.
|
||
Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently. | ||
In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. | ||
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | |
This means that the [*managed memory*](../memory/mem_setup.html#managed-memory) configuration affects |
Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently. | ||
In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. | ||
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | ||
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | |
the performance of your applications. Flink will use as much [managed memory](../memory/mem_setup.html#managed-memory) |
In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. | ||
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | ||
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | ||
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely | |
as configured for the execution of batch jobs. This prevents `OutOfMemoryErrors` because Flink knows precisely |
This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects | ||
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | ||
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely | ||
how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient, | |
how much memory it can use. If the [*managed memory*](../memory/mem_setup.html#managed-memory) is not large enough, |
on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory) | ||
as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely | ||
how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient, | ||
Flink will gracefully spill to disk. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Flink will gracefully spill to disk. | |
then Flink will gracefully spill to disk. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Last round of review. Thanks a lot for creating this documentation @azagrebin. I had a couple of comments which we can resolve before merging this PR. Once this is done +1 for merging.
docs/ops/memory/mem_trouble.md
Outdated
## IllegalConfigurationException | ||
|
||
If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates | ||
that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.) | |
that there is either an invalid configuration value (e.g. negative memory size, fraction that is greater than 1, etc.) |
docs/ops/memory/mem_trouble.md
Outdated
|
||
## OutOfMemoryError: Java heap space | ||
|
||
The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size | |
The exception usually indicates that the JVM heap is too small. You can try to increase the JVM heap size |
docs/ops/memory/mem_trouble.md
Outdated
## OutOfMemoryError: Java heap space | ||
|
||
The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size | ||
by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory). | |
by increasing [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory). |
docs/ops/memory/mem_trouble.md
Outdated
The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size | ||
by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory). | ||
|
||
<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option | |
<span class="label label-info">Note</span> You can also increase the [framework heap memory](mem_setup.html#framework-memory) but this option |
docs/ops/memory/mem_trouble.md
Outdated
by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory). | ||
|
||
<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option | ||
is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory. | |
is advanced and should only be changed if you are sure that the Flink framework itself needs more memory. |
docs/ops/memory/mem_migration.md
Outdated
Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks | ||
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)), | ||
see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory). | ||
The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend) | |
The JVM heap memory is also used by the heap state backends ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend) |
docs/ops/memory/mem_migration.md
Outdated
|
||
See also [how to configure managed memory now](mem_setup.html#managed-memory). | ||
|
||
### Explicit size |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Capitalization
|
||
If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job, | ||
its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory). | ||
The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we could add that one could turn this behaviour off if one wants the old behaviour.
docs/ops/memory/mem_migration.md
Outdated
## Container Cut-Off Memory | ||
|
||
For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations. | ||
Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc. | |
Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM, etc. |
docs/ops/memory/mem_migration.md
Outdated
The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory). | ||
See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory). | ||
|
||
The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options: | |
The other direct or native off-heap memory consumers can now be addressed by the following new configuration options: |
e34dafa
to
0cb8834
Compare
Thanks for the review @tillrohrmann |
What is the purpose of the change
In release 1.10, with FLIP-49, we introduced significant changes to the TaskExecutor memory model and it's related configuration options / logics.
It is very important that we clearly state the changes and potential effects, and guide our users to tune their clusters with the new configuration for both new setups and migrations of previous setups.
Brief change log
Verifying this change
open
localhost:4000
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (no)Documentation